The exemplary embodiment relates to text recognition in images. It finds particular application in connection with recognizing license plates and will be described with particular reference thereto. However, it is to be appreciated that it is applicable to a wide range of recognition problems.
Text recognition in images involves both recognizing that a portion of the image contains text and also recognizing the character sequence which constitutes the text. There are many instances where it is desirable to recognize text in images, for example, recognition of license plate numbers in images of vehicles, recognition of street names on images of street scenes, and the like. It may also be desirable to recognize different types of text (e.g., typed text vs. handwritten text) and to recognize different types of images (e.g., natural images vs. document images).
Recognition of license plate information assists in vehicle recognition, since in general the license plate is a unique identifier for the vehicle on which it is mounted. In the past, this problem has been traditionally addressed by applying Optical Character Recognition (OCR) on the license plate image. See, for example, Anagnostopoulos, et al., “License plate recognition from still images and video sequences: A survey,” IEEE Trans. on Intelligent Transportation Systems, vol. 9, No. 3, pp. 377-391, 2008, hereinafter “Anagnostopoulos”). However, OCR recognition can be computationally expensive and accuracy diminishes when the visibility at the time of capturing the image is poor.
A recent solution has been to address recognition as an image matching problem, as disclosed, for example, in copending U.S. application Ser. No. 13/300,124, filed on Nov. 18, 2011, by Jose Antonio Rodriguez-Serrano et al., entitled “METHODS AND SYSTEMS FOR IMPROVED LICENSE PLATE SIGNATURE MATCHING BY SIMILARITY LEARNING ON SYNTHETIC IMAGES”. Given an image of a license plate (the query), the license plate number of the closest matching images in a large database are retrieved. The images to be compared are each represented by an image signature, which is a statistical representation of an image, derived from low-level features extracted from the image. As image signatures, Fisher Vectors can be used. See, for example. Perronnin, et al., “Improving the Fisher kernel for large-scale image classification,” in ECCV, 2010.
The signature comparison method assumes that at least one example of the query is already present in the database. While this is often not an issue in some applications (for example, in the context of a parking application where an image taken at the exit is being matched to images taken at the entry), there are many instances where such a database is not available or is incomplete. One way that this could be addressed is by generating artificial license plates. For example, U.S. patent application Ser. No. 13/224,373, filed on Sep. 2, 2011, by Jose Antonio Rodriguez Serrano, et al., entitled “TEXT-BASED SEARCHING OF IMAGE DATA”, discloses a method for creation of virtual license plates by combining similar license plates. U.S. patent application Ser. No. 13/300,124, filed on Nov. 18, 2011, by Jose Antonio Rodriguez-Serrano et al., entitled “METHODS AND SYSTEMS FOR IMPROVED LICENSE PLATE SIGNATURE MATCHING BY SIMILARITY LEARNING ON SYNTHETIC IMAGES” and U.S. patent application Ser. No. 13/458,464, filed Apr. 27, 2012, by Jose Antonio Rodriguez-Serrano et al., “ENTITLED METHODS AND SYSTEMS FOR IMPROVING YIELD IN WANTED VEHICLE SEARCHES”, disclose methods for synthesizing license plate images.
The first of these methods is focused on retrieval and yields good results in terms of accuracy when the goal is to ensure that the license plate will likely be among the most similar retrieved images (e.g., among the top 20). This is generally sufficient for manually assisted search applications, but can pose problems for recognition, where usually a high top-1 accuracy is desired, i.e., it is desired to identify a single match with a high degree of accuracy, where a match is actually present. The second method can generate photo-realistic images of license plates from a given sequence of characters. However, it relies on a certain prior knowledge of the domain of application (e.g., license plate background, font, and the like). Additionally, multiple images are typically generated with different transformations to account for a set of representative plate distortions, which can be computationally expensive.
The disclosed method and system of performing text-to-image queries finds application in the context of vehicle re-identification systems, such as systems to automatically manage vehicle entries and exits in parking lots. Specifically, vehicle re-identification systems that capture and analyze a license plate image associated with a vehicle. A conventional approach to identify a vehicle is to extract a license plate number using an Automatic License Plate Recognition (ALPR) system and generating an exact license plate number lookup database including all plate numbers of entered vehicles.
This approach to re-identification has two main limitations:
First, the accuracy of ALPR systems is limited. Despite the claims of some ALPR providers, the re-identification accuracy of performing ALPR+exact string matching has been measured to be in the range 85%-90% in real production environments where images are captured from managed vehicle parking lots using existing image capture and ALPR equipment. Furthermore, the re-identification accuracy of performing ALPR and exact string matching has been measured to be as low as 70% in challenging conditions such as outdoor car parks with non-frontal cameras.
Second, recognizing and storing any license plate number can pose a privacy concern, as it explicitly reveals a nominal piece of information, i.e., a license plate number is considered personal information in many countries including France.
In U.S. patent application Ser. No. 14/054,998, filed Oct. 16, 2013, by RODRÍGUEZ-SERRANO et al., entitled “DELAYED VEHICLE IDENTIFICATION FOR PRIVACY ENFORCEMENT”, disclosed is a system to perform re-identification based on image signature matching, which can obtain higher accuracy results than commercial ALPR systems and does not pose privacy concerns. U.S. patent application Ser. No. 14/497,417, filed Sep. 26, 2014, by Rodriguez-Serrano et al., entitled “MULTI-QUERY PRIVACY-PRESERVING PARKING MANAGEMENT SYSTEM AND METHOD”, extends the system disclosed in U.S. patent application Ser. No. 14/054,998, filed Oct. 16, 2013, by RODRÍGUEZ-SERRANO et al., entitled “DELAYED VEHICLE IDENTIFICATION FOR PRIVACY ENFORCEMENT”, to allow the retrieval of license plate images from query strings, i.e. a user types in a license plate number and images of the corresponding license plate are returned. This enables, among other operations, to recover the entry time of users with lost tickets, to perform security checks, or to enable “where is my car” searches.
In addition to a system and method for recognizing text in images which is both sufficiently accurate for a particular application and computationally efficient, needed is a method and system to enable text queries with wildcards symbols, e.g., queries of the form ?BC12?, where the symbol ? stands for “any character”, thus, for example, ABC123 and ZBC127 would match the query. The ability to handle text queries with wildcard symbols is necessary because in many practical situations the issuer of the query does not know the subject license plate in its entirety. While such a query with wildcards is possible in ALPR based systems, where a database of possible license plate numbers is maintained, text-to-image matching does not currently provide the ability to handle, efficiently, text queries with wildcard symbols.
Provided herein is a method and system to perform text-to-image matching with or without wildcards, e.g., retrieval of license plate images from string queries with wildcards.